GESTURE, SPEECH, AND LEXICAL ACCESS: The Role of Lexical Movements in Speech Production

In a within-subjects design that varied whether speakers were allowed to gesture and the difficulty of lexical access, speakers were videotaped as they described animated action cartoons to a listener. When speakers were permitted to gesture, they gestured more often during phrases with spatial content than during phrases with other content. Speech with spatial content was less fluent when speakers could not gesture than when they couid gesture; speech with nonspatial content was nol affected by gesture condition. Preventing gesturing increased the relative frequency of nonjuncture filled pauses in speech with spatial content, but not in speech with other content. Overall, the effects of preventing speakers from gesturing resembled those of increasing the difficulty of lexical access by other means, except that the effects of gesture restriction were specific to speech with spatial content. The findings support the hypothesis that gestural accompaniments to spontaneous speech can facilitate access to the mental lexicon. Conversational gestures are unplanned, fluent hand niovements that often accompany spontaneous speech. The prevailing view is that they enhance communication by conveying information that amplifies and modulates information conveyed in the speech channel (Birdwhistell, 1970; Graham & Argyle, 1975; Kendon, 1983, 1987), However, recent research casts doubt on the communicative importance of conversational gestures, at least insofar as the semantic information they convey is concerned (Feyereisen, Van de Wiele, & Dubois, 1988; Krauss, Dushay, Chen, & Rauscher, 1995; Krauss, Morrel-Samuels, & Colasante, 1991), leading several investigators to speculate about possible noncommunicative functions they may serve (Feyereisen & deLannoy, 1991; Hadar, 1989; Krauss et al., 1991; MorrelSamuels & Krauss, 1992; Rime & Schiaratura, 1991), One possibility is that conversational gestures are generated as part of the speech production process, and play a role in the retrieval of words from lexical memory. The general idea that gestures enhance a speaker's ability to access obscure or unfamiliar words is not a new one, having been proposed by a remarkably varied assortment of writers over the past 60 years (DeLaguna, 1927; Dobrogaev, 1929; Ekman & Friesen, 1972; Freedman, 1972; Mead, 1934; Moscovici, 1%7; Werner & Kaplan, 1963), The earliest empirical study is probably that of Dobrogaev, who asked speakers to ctirb facial expression, gestures, and head movements while speaking. He reported that this restriction resulted in decreased fluency, impaired articuAddress correspondence to Frances H, Rauscher, Department of Psychology, University of Wisconsin, Oshkosh, WI54901; or to Robert M. Krauss, Departtnent of Psychology, Colutnbia University, New York, NY 10027; e-mail: [email protected],edu. lation, and reduced vocabulary size. Unfortunately, the article, like many written in that era, provides little in the way of procedural details and describes results in an impressionistic, nonquantitative fashion. More recently, three studies have examined the effects of preventing gesturing on speech, Lickiss and Wellens (1978) found no effects of restraining speakers' hand movements on verbal fluency, but it is unclear exactly which dysfluencies were examined. Using a varied set of linguistic indices, Graham and Heywood (1975) examined the speech of 6 subjects describing abstract line drawings on trials during which they were permitted to gesture and trials during which they were not. Although statistically signiflcant effects of preventing gesturing were found on some of the indices, Graham and Heywood concluded that "elimination of gesture has no particularly marked effects on speech performance" (p, 194). Given their small sample of speakers and the fact that statistically significant or nearsignificant effects were found for several contrasts, failure to reject the null hypothesis seems a weak justification for so strong a conclusion. In a rather different sort of study. Rime, Schiaratura, Hupet, and Ghysselinckx (1984) compared the content of subjects' speech while their heads, arms, hands, legs, and feet were restrained with the content of their speech under normal circumstances. Less vivid imagery was observed when speakers could not move. In considering functions gestures might serve, it is useful to distinguish between two different types of what we call conversational gestures.' Although gestural typologies abound in the literature, virtually all researchers recognize a category of conversational gestures that are simple, brief, repetitive, coordinated with the speech prosody, and apparently unrelated in form to the conceptual content of the speech they accompany. We refer to such gestures as motor movements {Hadar, 1989). They also have been called "batonic gestures" (BuU, 1987) and "beats" (McNeill, 1987, 1992), A second category of conversational gestures consists of movements that are more complex, less repetitive, more varied, and of longer duration than motor movements and that seem related in form to the ideational content of the accompanying speech. We refer to this second category of gestures as lexical movements (Hadar, 1989), They also have been called "illustrators" (Ekman & Friesen, 1972) and "representational gestures" (McNeill, 1992). Our central hypothesis is that such gestures play a role in lexical access. How might gesturing affect speech? Speech production begins with the formulation of a commtuiicative intention—a con1, Conversational gestures differ from the gestural hand signs with conventionalized meanings (e.g., "thumbs-up," "A-OK") often referred to as emblems. Emblems can be used in the absence of speech and clearly convey semantic information. 226 Copyright © 1996 American Psychologica! Society VOL. 7, NO. 4, JULY 19% PSYCHOLOGICAL SCIENCE Frances H, Rauscher, Robert M, Krauss, and Yihsiu Chen ceptual structure to be conveyed by the utterance,^ Levelt (1989) calls this s t^e of the process conceptualizing and refers to its output as a preverbal message. The preverbal message specifies the semantic features to be used in lexical selection, and in constructing it the speaker may draw upon knowledge represented in memory in any of a number of representational formats. In the second stage of the speech production process, which Levelt cails formulating, the preverbal message is transformed into a linguistic structtire. As part of this process, elements of the preverbal messE^e activate entries in the mental lexicon, permitting the speaker to select lexical items that satisfy the previously determined semantic specifications. The output of this stage is a surface structure, which is then further processed by a phonological encoder into a set of phonetic instructions or an articulatory plan. We believe that lexical movements derive from knowledge that is encoded in a spatial format, and that the spatial features of the conceptual structure that are expressed in a lexical movement facilitate lexical access by cross-modally priming the semantic features that enter into lexical search.' If lexical movements facilitate lexical access, preventing speakers from gesturing should make lexical access more difficult. Problems in lexical access often are reflected in slow, dysfluent speech. For example, unpredictable (hence, less accessible) words in spontaneous speech tend to be preceded by silent and filled pauses (Goldman-Eisier, 1958; Tannenbaum, Williams, & Hiller, 1965), The rate of filled pauses in a speech corpus is positively correlated with its lexical diversity, an indication of the range of alternatives from which lexical selection is made (Schachter, Christenfeld, Ravina, & Bilous, 1991), Artificially increasing the difficulty of lexical access (e,g., by asking the speaker to avoid using words that contain a particular letter) increases the frequency of filled pauses (Boomer & Dittmann, 1964), Of course, not all pauses, filled or silent, reflect problems in lexical access. From time to time, speakers must pause to breathe. They also pause to plan the ideas they want to convey (Butterworth, 1980), In addition, speakers may insert pauses for communicative reasons—as a kind of audible punctuation to guide listeners' comprehension. In spontaneous speech, about 60% to 70% of the pauses fall at grammatical clause boundaries (often called juncture pauses), but speech read from text (for which neither planning nor lexical access is problematic) contains many fewer pauses, with nearly all of them falling at junctures (Henderson, Goldman-Eisier, & Skarbek, 1965). Although it is difficult to determine a juncture pause's origin, pauses that fall within grammatical clauses (often called hesitations or nonjuncture pauses) typically reflect problems in 2. Our description of the speech production process is based on the model proposed by Levelt (1989). The description omits many details of what is an extremely complex process, and many of these details are matters of considerable contention. However, virtually all of the speech production models that have been proposed draw a distinction between conceptualizing and formulating stages of the process. 3, We also believe that conceptual contents encoded in other formats (e.g., motoric) can be represented gesturally, but the data we present concern oniy spatial content. For a more detailed exposition of this model, see Krauss, Chen, and Chawla (1996). speech production. One of the effects of rehearsing speech material is to increase the proportion of pauses that fall at grammatical junctures (Butterworth, 1980). In one study (Chawla & Krauss, 1994), about 72% of the filled and silent pauses in rehearsed narratives fell at grammatical junctures; in the spontaneous versions of these same narratives, the rate was 40%, If our conjecture about the function of conversational gestures is correct, preventing speakers from gesturing should increase the relative frequency of nonjuncture pauses and other dysfluencies. In the experiment reported here, we examined the effects of preventing speakers from gesturing as they described animated action cartoons to a listener. We also artificially manipulated the difficulty of lexical access by constraining what they could say. We expected that the effects of preventing speakers from gesturing would parallel those of constraining their speech.